<!DOCTYPE html>


<html lang="zh-CN">


<head>
  <meta charset="utf-8" />
    
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>
    ceph简介及高可用集群搭建.md |  
  </title>
  <meta name="generator" content="hexo-theme-ayer">
  
  <link rel="shortcut icon" href="/favicon.ico" />
  
  
<link rel="stylesheet" href="/dist/main.css">

  
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css">

  
<link rel="stylesheet" href="/css/custom.css">

  
  
<script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>

  
  

  

</head>

</html>

<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-deploy/ceph简介及高可用集群搭建"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  ceph简介及高可用集群搭建.md
</h1>
 

    </header>
     
    <div class="article-meta">
      <a href="/2020/11/11/deploy/ceph%E7%AE%80%E4%BB%8B%E5%8F%8A%E9%AB%98%E5%8F%AF%E7%94%A8%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA/" class="article-date">
  <time datetime="2020-11-10T16:00:00.000Z" itemprop="datePublished">2020-11-11</time>
</a> 
  <div class="article-category">
    <a class="article-category-link" href="/categories/deploy/">deploy</a>
  </div>
  
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">2.3k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">10 分钟</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <h1 id="ceph简介及高可用集群搭建"><a href="#ceph简介及高可用集群搭建" class="headerlink" title="ceph简介及高可用集群搭建"></a>ceph简介及高可用集群搭建</h1><p>Ceph简介</p>
<p>Ceph是一个PB，EB级别的分布式存储系统，可以提供文件存储，对象存储、和块存储，它可靠性高，易扩展，管理简便，其中对象存储和块存储可以和其他云平台集成。一个Ceph集群中有Monitor节点、MDS节点（用于文件存储）、OSD守护进程。<br>Ceph组件</p>
<p>Ceph OSD：OSD守护进程，用于存储数据、数据恢复、数据回滚、平衡数据，并通过心跳程序向Monitor提供数据相关的监控信息。一个Ceph集群中至少需要两个OSD守护进程，这两个osd需要在两个物理服务器上，而不是在一台服务器上启动两个osd。</p>
<p>Monitor：维护集群状态的映射信息，包括monmap，osdmap，pgmap；至少需要三个monitor来实现高可用</p>
<p>MDS：存储Ceph文件系统的元数据。</p>
<p>Managers：Ceph Manager守护程序（ceph-mgr）负责跟踪运行时指标和Ceph集群的当前状态，包括存储利用率，当前性能指标和系统负载。CephManager守护进程还托管基于python的模块来管理和公开Ceph集群信息，包括基于Web的Ceph Dashboard和RESTAPI。高可用性通常至少需要两个<br>安装环境准备</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br></pre></td><td class="code"><pre><span class="line">准备机器，osd节点机器需要两个硬盘，配置4GiB/4vCPU/60G x2</span><br><span class="line">监控节点：</span><br><span class="line">monitor1：192.168.85.128</span><br><span class="line">monitor2：192.168.85.130</span><br><span class="line">monitor3：192.168.85.131</span><br><span class="line">osd节点：</span><br><span class="line">osd1：192.168.85.133</span><br><span class="line">osd2：192.168.85.134</span><br><span class="line">初始化机器</span><br><span class="line"></span><br><span class="line">1.修改主机名</span><br><span class="line">在monitor1上操作：</span><br><span class="line">hostnamectl set-hostname monitor1</span><br><span class="line">在monitor2上操作：</span><br><span class="line">hostnamectl set-hostname monitor2</span><br><span class="line">在monitor3上操作：</span><br><span class="line">hostnamectl set-hostname monitor3</span><br><span class="line">在osd1上操作：</span><br><span class="line">hostnamectl set-hostname osd1</span><br><span class="line">在osd2上操作：</span><br><span class="line">hostnamectl set-hostname osd2</span><br><span class="line">2.修改host文件，在各个节点操作</span><br><span class="line">修改各个主机节点hosts文件</span><br><span class="line">/etc/hosts文件如下</span><br><span class="line">192.168.85.128 monitor1</span><br><span class="line">192.168.85.130 monitor2</span><br><span class="line">192.168.85.131 monitor3</span><br><span class="line">192.168.85.133 osd1</span><br><span class="line">192.168.85.134 osd2</span><br><span class="line">3.配置master1到三个节点无密码登陆，在master1上操作</span><br><span class="line">ssh-keygen -t rsa   #一直回车就可以</span><br><span class="line">ssh-copy-id -i .ssh/id_rsa.pub monitor1</span><br><span class="line">ssh-copy-id -i .ssh/id_rsa.pub monitor2</span><br><span class="line">ssh-copy-id -i .ssh/id_rsa.pub monitor3</span><br><span class="line">ssh-copy-id -i .ssh/id_rsa.pub osd1</span><br><span class="line">ssh-copy-id -i .ssh/id_rsa.pub osd2</span><br><span class="line">4.初始化机器（各个节点操作）</span><br><span class="line">yum -y install wget net-tools nfs-utils lrzsz gccgcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntplibaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-develepel-release lrzsz openssh-server</span><br><span class="line">yum install deltarpm -y</span><br><span class="line">5.关闭firewalld防火墙（各个节点操作）</span><br><span class="line">停止firewalld服务 停止防火墙，并禁用这个服务</span><br><span class="line">systemctl stop firewalld.service &amp;&amp; systemctldisable firewalld.service</span><br><span class="line">安装iptables 如果你要改用iptables的话，需要安装iptables服务</span><br><span class="line"><span class="meta">#</span><span class="bash">安装</span></span><br><span class="line">yum install iptables-services -y</span><br><span class="line">禁用iptables</span><br><span class="line">service iptables stop &amp;&amp; systemctl disableiptables</span><br><span class="line">6.时间同步</span><br><span class="line">在master1上</span><br><span class="line">ntpdate cn.pool.ntp.org</span><br><span class="line">systemctl start ntpd &amp;&amp;systemctl enable ntpd</span><br><span class="line">在除了master1的其他节点上</span><br><span class="line">ntpdate monitor1</span><br><span class="line">计划任务：</span><br><span class="line">* */1 * * * /usr/sbin/ntpdate monitor1</span><br><span class="line">安装ceph-deploy</span><br><span class="line"></span><br><span class="line">1.安装epel源，在各个节点上操作</span><br><span class="line">yum install -y yum-utils&amp;&amp; sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/&amp;&amp; sudo yum install --nogpgcheck -y epel-release &amp;&amp; sudo rpm--import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 &amp;&amp; sudo rm/etc/yum.repos.d/dl.fedoraproject.org*</span><br><span class="line">2.配置ceph的yum源，在各个节点上操作</span><br><span class="line">cat /etc/yum.repos.d/ceph.repo</span><br><span class="line">[Ceph]</span><br><span class="line">name=Ceph packages for$basearch</span><br><span class="line">baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/</span><br><span class="line">enabled=1</span><br><span class="line">gpgcheck=0</span><br><span class="line">type=rpm-md</span><br><span class="line">gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc</span><br><span class="line">priority=1</span><br><span class="line">[Ceph-noarch]</span><br><span class="line">name=Ceph noarch packages</span><br><span class="line">baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/</span><br><span class="line">enabled=1</span><br><span class="line">gpgcheck=0</span><br><span class="line">type=rpm-md</span><br><span class="line">gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc</span><br><span class="line">priority=1</span><br><span class="line">[ceph-source]</span><br><span class="line">name=Ceph source packages</span><br><span class="line">baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS/</span><br><span class="line">enabled=1</span><br><span class="line">gpgcheck=0</span><br><span class="line">type=rpm-md</span><br><span class="line">gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc</span><br><span class="line">priority=1</span><br><span class="line">3.安装ceph-deploy</span><br><span class="line">yum update -y        #在各个节点上操作</span><br><span class="line">yum install ceph-deploy ceph -y     #在master1上操作</span><br><span class="line">yum installyum-plugin-priorities -y #在master1上操作</span><br><span class="line">在除了master1的其他节点上操作</span><br><span class="line">yum install ceph -y</span><br><span class="line">4.创建ceph monitor</span><br><span class="line">（1）创建一个目录，用于保存ceph-deploy 生成的配置文件和密钥信息的，在monitor1上操作，</span><br><span class="line">mkdir /root/ceph-deploy&amp;&amp; cd /root/ceph-deploy</span><br><span class="line">（2） 初始化monitor节点，在monitor1上操作，</span><br><span class="line">ceph-deploy new monitor1 monitor2 monitor3</span><br><span class="line">ls可发现在ceph-deploy目录下多了几个文件，如下面这些</span><br><span class="line">ceph.conf ceph-deploy-ceph.log ceph.mon.keyring</span><br><span class="line">ceph.conf            #ceph集群配置文件</span><br><span class="line">ceph-deploy-ceph.log #ceph-deploy命令执行结果的日志</span><br><span class="line">ceph.mon.keyring     #ceph mon角色的key</span><br><span class="line">（3）修改ceph配置文件，在各个monitor节点操作</span><br><span class="line">cat ceph.conf</span><br><span class="line">[global]</span><br><span class="line">fsid = dc97ea52-e32c-404d-9bbc-9d0a90e39e02</span><br><span class="line">mon_initial_members = monitor1, monitor2,monitor3</span><br><span class="line">mon_host =192.168.85.128,192.168.85.129,192.168.85.130</span><br><span class="line">auth_cluster_required = cephx</span><br><span class="line">auth_service_required = cephx</span><br><span class="line">auth_client_required = cephx</span><br><span class="line">osd_pool_default_size = 2</span><br><span class="line">因为我们环境中有两个OSD，Ceph模式的默认osd副本个数为3，因此需要修改配置文件Ceph.conf，在[global]部分增加如下配置：</span><br><span class="line">osd pool default size = 2</span><br><span class="line">（4）安装ceph集群，在monitor1上操作</span><br><span class="line">ceph-deploy install monitor1 monitor2monitor3 osd1 osd2</span><br><span class="line">上面使用ceph-deploy来安装ceph程序，我们之前通过yum手动在各个节点安装过ceph，为了保证ceph安装成功，这两种方法都需要使用</span><br><span class="line">（5）部署初始监视器并收集密钥，在monitor1上操作</span><br><span class="line">ceph-deploy mon create-initial</span><br><span class="line">ll /root/ceph-deploy 会看到生成很多密钥</span><br><span class="line">ceph.bootstrap-mds.keyring</span><br><span class="line">ceph.bootstrap-mgr.keyring</span><br><span class="line">ceph.bootstrap-osd.keyring</span><br><span class="line">ceph.bootstrap-rgw.keyring</span><br><span class="line">ceph.client.admin.keyring</span><br><span class="line">ceph.bootstrap-mds.keyring   #MDS启动key</span><br><span class="line">ceph.bootstrap-osd.keyring   #OSD启动key </span><br><span class="line">ceph.client.admin.keyring    #管理员key</span><br><span class="line">（6）将配置文件和管理密钥分发到各个ceph节点，在monitor1上操作</span><br><span class="line">ceph-deploy admin monitor1 monitor2 monitor3 osd1 osd2</span><br><span class="line">（7）授权，在各个节点操作</span><br><span class="line">chmod +r/etc/ceph/ceph.client.admin.keyring</span><br><span class="line">配置osd，下面步骤均在monitor1上操作</span><br><span class="line"></span><br><span class="line">1.给ceph添osd，并完成激活</span><br><span class="line">在osd节点osd1和osd2上操作</span><br><span class="line">mkdir /data/osd1 -p</span><br><span class="line">chmod 777 /data/osd1/</span><br><span class="line">chown -R ceph.ceph /data/osd1</span><br><span class="line">2.回到master1上操作，准备osd</span><br><span class="line">ceph-deploy osd prepare osd1:/data/osd1osd2:/data/osd1</span><br><span class="line">3.激活osd：</span><br><span class="line">ceph-deploy osd activate osd1:/data/osd1osd2:/data/osd1</span><br><span class="line">ceph health   #查看ceph集群状态，显示如下，在health处看到HEALTH_OK说明集群正常</span><br><span class="line">    cluster a53e18df-5c06-46bb-b4db-7c44f64a367e</span><br><span class="line">     health HEALTH_OK</span><br><span class="line">     monmap e1: 3 mons at &#123;monitor1=192.168.85.128:6789/0,monitor2=192.168.85.130:6789/0,monitor3=192.168.85.131:6789/0&#125;</span><br><span class="line">            election epoch 6, quorum 0,1,2 monitor1,monitor2,monitor3</span><br><span class="line">     osdmap e14: 2 osds: 2 up, 2 in</span><br><span class="line">            flags sortbitwise,require_jewel_osds</span><br><span class="line">      pgmap v618: 320 pgs, 2 pools, 14752 kB data, 23 objects</span><br><span class="line">            14146 MB used, 64246 MB / 78393 MB avail</span><br><span class="line">                 320 active+clean</span><br><span class="line">4.创建ceph pool池</span><br><span class="line">ceph osd pool create testpool 256</span><br><span class="line">5.创建rbd</span><br><span class="line">rbd create testpool/myrbd --size 10240</span><br><span class="line">6.映射块设备到自己机器</span><br><span class="line">rbd feature disable testpool/myrbd object-map fast-diff deep-flatten</span><br><span class="line"><span class="meta">#</span><span class="bash">禁用一些map，否则客户端挂载不成功</span></span><br><span class="line">rbd map testpool/myrbd </span><br><span class="line">显示/dev/rbd0，说明映射成功</span><br><span class="line">7.测试客户端挂载使用</span><br><span class="line">mkdir /mnt/firstrbd </span><br><span class="line">mkfs.xfs /dev/rbd0 </span><br><span class="line">mount /dev/rbd0 /mnt/firstrbd</span><br><span class="line">8.kubernetes对接ceph</span><br><span class="line">可看视频了解，视频链接如下：</span><br><span class="line">https://edu.51cto.com/sd/d7dff</span><br><span class="line">常见错误排查</span><br><span class="line"></span><br><span class="line">1.too many PGs per OSD</span><br><span class="line">上面测试客户端挂载之后，我们在monitor1上用ceph -s再次查看集群状态</span><br><span class="line">ceph -s</span><br><span class="line">可看到有警告health HEALTH_WARN</span><br><span class="line">                     too many PGs per OSD (320 &gt; max300)</span><br><span class="line">clustera53e18df-5c06-46bb-b4db-7c44f64a367e</span><br><span class="line">     health HEALTH_WARN</span><br><span class="line">            too many PGs per OSD (320 &gt; max300)</span><br><span class="line">     monmap e1: 3 mons at &#123;monitor1=192.168.85.128:6789/0,monitor2=192.168.85.130:6789/0,monitor3=192.168.85.131:6789/0&#125;</span><br><span class="line">            election epoch 4, quorum 0,1,2monitor1,monitor2,monitor3</span><br><span class="line">     osdmap e14: 2 osds: 2 up, 2 in</span><br><span class="line">            flagssortbitwise,require_jewel_osds</span><br><span class="line">      pgmap v253: 320 pgs, 2 pools, 14752 kBdata, 23 objects</span><br><span class="line">            14158 MB used, 64235 MB / 78393 MBavail</span><br><span class="line">                 320 active+clean</span><br><span class="line">这个警告是因为ceph集群中osd 数量较少，创建了pool，每个pool要占用一些pg ，ceph集群默认每个osd 能有300个pgs，</span><br><span class="line">这个默认值是可以调整的，但调整会对集群的性能产生一定影响。如果是测试环境，可以消除这种警告，当然这种警告也可以不用理会。</span><br><span class="line">查询每个osd下最大的pg阈值：</span><br><span class="line">ceph --show-config | grep mon_pg_warn_max_per_osd</span><br><span class="line"><span class="meta">#</span><span class="bash">可以看到每个osd默认有300个pg</span></span><br><span class="line">mon_pg_warn_max_per_osd = 300</span><br><span class="line">解决方法如下：</span><br><span class="line">在monitor1，monitor2，monitor3上修改ceph配置文件，把osd上的pg数量修改成600，cat /etc/ceph/ceph.conf</span><br><span class="line">[global]</span><br><span class="line">...</span><br><span class="line">mon_pg_warn_max_per_osd = 600</span><br><span class="line">上面修改好之后重启monitor服务，在monitor1上操作</span><br><span class="line">systemctl restart ceph-mon@monitor1</span><br><span class="line">systemctl restart ceph-mon@monitor2</span><br><span class="line">systemctl restart ceph-mon@monitor3</span><br><span class="line">ceph -s    看到下面的ceph集群状态正常了</span><br><span class="line">    cluster a53e18df-5c06-46bb-b4db-7c44f64a367e</span><br><span class="line">     health HEALTH_OK</span><br><span class="line">     monmap e1: 3 mons at &#123;monitor1=192.168.85.128:6789/0,monitor2=192.168.85.130:6789/0,monitor3=192.168.85.131:6789/0&#125;</span><br><span class="line">            election epoch 6, quorum 0,1,2 monitor1,monitor2,monitor3</span><br><span class="line">     osdmap e14: 2 osds: 2 up, 2 in</span><br><span class="line">            flags sortbitwise,require_jewel_osds</span><br><span class="line">      pgmap v618: 320 pgs, 2 pools, 14752 kB data, 23 objects</span><br><span class="line">            14146 MB used, 64246 MB / 78393 MB avail</span><br><span class="line">                 320 active+clean</span><br><span class="line">2.重启monitor服务报错</span><br><span class="line">systemctl restart ceph-mon@monitor1</span><br><span class="line">systemctl restart ceph-mon@monitor2</span><br><span class="line">systemctl restart ceph-mon@monitor3</span><br><span class="line">如果启动报错，按如下排查</span><br><span class="line">systemctl status ceph-mon@monitor1</span><br><span class="line">报错如下：</span><br><span class="line">Sep 05 22:55:30 monitor1 systemd[1]: start request repeated tooquickly for ceph-mon@monitor1.service</span><br><span class="line">Sep 05 22:55:30 monitor1 systemd[1]: Failed to start Ceph clustermonitor daemon.</span><br><span class="line">解决方法：</span><br><span class="line">在各个节点修改配置文件</span><br><span class="line">vim /usr/lib/systemd/system/ceph-mon@.service</span><br><span class="line">vim /usr/lib/systemd/system/ceph-mon@.service</span><br><span class="line">vim /usr/lib/systemd/system/ceph-mon@.service</span><br><span class="line">把StartLimitInterval=30min加上注释</span><br><span class="line">在各个节点执行如下，重新加载配置文件</span><br><span class="line">systemctl daemon-reload</span><br><span class="line">在monitor1上执行如下，重启，可启动成功</span><br><span class="line">systemctl restart ceph-mon@monitor1 </span><br><span class="line">systemctl restart ceph-mon@monitor2 </span><br><span class="line">systemctl restart ceph-mon@monitor3 </span><br><span class="line">3.全都重新部署</span><br><span class="line">当你在部署时遇到问题，排查过程困难重重的话，那么不妨用如下方法，把集群清空，重新部署，可以执行以下清除Ceph包之后重新开始安装：</span><br><span class="line">在monitor1上操作</span><br><span class="line">cd /root/ceph-deploy</span><br><span class="line">ceph-deploy purge monitor1 monitor2 monitor3 osd1 osd2</span><br><span class="line">ceph-deploy purgedata monitor1 monitor2 monitor3 osd1 osd2ceph-deployforgetkeys</span><br><span class="line">rm ceph.*</span><br></pre></td></tr></table></figure>

 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          打赏
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>版权声明： </strong>
          
          本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=http://example.com/2020/11/11/deploy/ceph%E7%AE%80%E4%BB%8B%E5%8F%8A%E9%AB%98%E5%8F%AF%E7%94%A8%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/deploy/" rel="tag">deploy</a></li></ul>

    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2020/11/11/deploy/Vagrant%E6%A8%A1%E6%9D%BF%E5%92%8C%E4%BD%BF%E7%94%A8/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            Vagrant模板和使用.md
          
        </div>
      </a>
    
    
      <a href="/2020/11/11/develop/GitHub%20%E4%B8%8A%E5%BC%80%E6%BA%90%E7%9A%84%20RBAC%20%E6%9D%83%E9%99%90%E7%AE%A1%E7%90%86%E7%B3%BB%E7%BB%9F/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">GitHub 上开源的 RBAC 权限管理系统.md</div>
      </a>
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "",
    app_key: "",
    path: window.location.pathname,
    avatar: "monsterid",
    placeholder: "给我的文章加点评论吧~",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
     
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2015-2020
        <i class="ri-heart-fill heart_icon"></i> TzWind
      </li>
    </ul>
    <ul>
      <li>
        
        
        
        由 <a href="https://hexo.io" target="_blank">Hexo</a> 强力驱动
        <span class="division">|</span>
        主题 - <a href="https://github.com/Shen-Yu/hexo-theme-ayer" target="_blank">Ayer</a>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>访问人数:<span id="busuanzi_value_site_uv"></span></s>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>浏览次数:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278069914&amp;web_id=1278069914'></script>
        
      </li>
    </ul>
  </div>
</footer>
      <div class="float_btns">
        <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

<div class="todark" id="todark">
  <i class="ri-moon-line"></i>
</div>

      </div>
    </main>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="Hexo"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">分类</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" target="_blank" rel="noopener" href="http://www.baidu.com">百度</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">友链</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/2019/about">关于我</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="搜索">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <script>
      if (window.matchMedia("(max-width: 768px)").matches) {
        document.querySelector('.content').classList.remove('on');
        document.querySelector('.sidebar').classList.remove('on');
      }
    </script>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯咖啡吧~</p>
  <div class="reward-box">
    
    
  </div>
</div>
    
<script src="/js/jquery-2.0.3.min.js"></script>


<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->


<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: '.tocbot',
    contentSelector: '.article-entry',
    headingSelector: 'h1, h2, h3, h4, h5, h6',
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: 'main',
    positionFixedSelector: '.tocbot',
    positionFixedClass: 'is-position-fixed',
    fixedSidebarOffset: 'auto'
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css">
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->

<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script>

<!-- MathJax -->

<!-- Katex -->

<!-- busuanzi  -->


<script src="/js/busuanzi-2.3.pure.min.js"></script>


<!-- ClickLove -->

<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->


<link rel="stylesheet" href="/css/clipboard.css">

<script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>


<!-- CanvasBackground -->


    
  </div>
</body>

</html>